Towards Better Domain Adaptation for Self-Supervised Models: A Case Study of Child ASR

Recently, self-supervised learning (SSL) from unlabelled speech data has gained increased attention in the automatic recognition (ASR) community. Typical SSL methods include autoregressive predictive coding (APC), Wav2vec2.0, and hidden unit BERT (HuBERT). However, models are biased to pretraining data. When finetuned with another domain, domain shifting occurs might cause limited knowledge transfer for downstream tasks. In this paper, we propose a novel framework, responsible adaptation finetuning (DRAFT), reduce pretrained models, evaluate it causal non-causal transformer. For transformer, an extension of APC (E-APC) is proposed learn richer information by using multiple temporally-shifted sequences perform prediction. various solutions bidirectional (Bi-APC) investigated. addition, DRAFT framework examined Wav2vec2.0 HuBERT methods, which use transformers as backbone. The experiments conducted on child ASR (using OGI MyST databases) trained adult Librispeech. relative WER improvements up 19.7% two tasks observed when compared without adaptation. With (E-APC DRAFT), even larger (30% 19% data, respectively) methods.